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Abstract. We demonstrate the possibility of coding parts, features that 
are higher level than boundaries, using a modified AT field after aug- 
menting the interaction term of the AT energy with a non-local term 
and weakening the separation into boundary /not-boundary phases. The 
iteratively extracted parts using the level curves with double point sin- 
gularities are organized as a proper binary tree. Inconsistencies due to 
non-generic configurations for level curves as well as due to visual changes 
such as occlusion are successfully handled once the tree is endowed with 
a probabilistic structure. The work is a step in establishing the AT func- 
tion as a bridge between low and high level visual processing. 

Keywords: phase fields, non-local variational shape analysis 

1 Introduction 

The phase field of Ambrosio and Tortorelli [1] (AT function) serving as a contin- 
uous indicator for the boundary/not-boundary state at every domain point has 
proven to be an indispensable tool in image and shape analysis. It is a minimizer 
of an energy composed of two competing terms: One term favors configurations 
that take values close to either or 1 (separation into boundary/not-boundary 
phases) and the other term encourages local interaction in the domain by penal- 
izing spatial inhomogeneity. A parameter controls the relative influence of these 
two terms, hence, the interaction. As this "interaction" parameter tends to 0, 
the separation term is strongly emphasized; consequently, the field tends to the 
characteristic function 1 — xs of the boundary set S and the AT energy tends 
(following the F convergence framework [4]) to the boundary length. 

In computer vision, the AT function first appeared as a technical device to ap- 
ply gradient descent to the Mumford-Shah functional [15]. Over the years, it has 
been extended in numerous ways to address a rich variety of visual applications. 
Earlier works include Shah and colleagues [22, 23, 25, 19], March and Dozio [13], 
Proesman, Pauwels and van Gool [21], Teboul et al. [29]. During the last couple 
of years we have witnessed an increasing number of promising works modifying 
or extending Ambrosio- Tortorelli/ Mumford-Shah based models. Some examples 
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are: Bar, Sochen and Kiryati [3], Rumpf and colleagues [6,20], Erdem, Sancar- 
Yilmaz and Tari [7], Patz and Preusser [17], Jung and Vese [9]. These works 
together with many others collaboratively established the role of AT function in 
variational formulations that jointly involve region and boundary terms. 

In the majority of the works, the AT function serves as an auxiliary vari- 
able to facilitate discontinuity-preserving smoothing and boundary detection. 
Relatedly, the interaction parameter is chosen sufficiently small to better local- 
ize boundaries. In contrast. Shah and Tari, starting with [27, 28] in late 90's, 
have focused on the ability of the AT function in coding morphologic proper- 
ties of shapes, regions construed by boundaries. Relatedly, they have weakened 
boundary/not-boundary separation either by choosing a large interaction pa- 
rameter or by other means [2] and focused on the geometric properties of the 
level curves after constructing the AT function (reviewed in [24]) for shapes as: 



where i? G is a bounded open set with a boundary dQ (denoting a shape); 
Xi7(x) is the shape indicator function which attains 1 in i? and on dQ] p is 
the parameter. The first term forces strong boundary /interior separation while 
the second one forces smoothness. 

The AT function of shape is related to a variety of morphological concepts. 
For instance, it is a weighted distance transform [11,12] with its level curves 
approximating curvature-dependent motion [16,10]. Thus, it enables extraction 
of local symmetries and skeletons directly from grayscale images; that is, it 
bridges image segmentation and shape description. The ability of level curves in 
coding morphological information is also exploited by Droske and Rumpf [20] to 
measure equivalence of two shapes in a registration problem. 

In this paper, following Shah and Tari [27, 28, 2], we explore and extend the 
ability of an AT-like field in coding features that are at a higher level than 
boundaries. Whereas the previous works focus on local symmetry axes, we focus 
on shape's intuitive components as coded via upper and lower level sets. Our 
constructions are based on a new field obtained as the minimizer of a modified 
AT energy. We discuss the geometry of the level curves of the new minimizer and 
exploit it to extract a part hierarchy tree endowed with a probabilistic structure. 

The considered modification involves an additive augmentation of the inter- 
action term with a non-local term in a way that the upper and lower zero level 
sets of the minimizer yield disjoint domains [26] within which the minimizer is 
morphologically equivalent to the AT function. Following the pioneering work of 
Buades, Coll and Morel [5], UCLA group formulated interesting non-local vari- 
ational formulations, including non-local versions of the Ambrosio-Tortorelli/ 
Shah approximations of the Mumford-Shah functional [9, 8] by replacing local 
image derivatives with non-local ones. This kind of modification is very different 
from our modification which modifies the phase field itself. 




^ boundary/interior local interaction 

separation 



with t'(x) = for x = (x, y) G df2 



(1) 
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In this paper, we focus on shapes. Nevertheless, the long term goal of our 
work is to bridge low level processes such as segmentation and image registration 
with the high level process of shape abstraction. Integration of the presented 
developments to Mumford-Shah type models via coupled PDEs framework is a 
future work. 

2 A Modified Energy and Its Minimizer 

Let us consider 



J I' i(c^(x) - /(x))2 + p [|V:^(x)|2 + (E^eacoi^)) 



dx dy 



with cu{x.) = for X = (x, y) G dn (2) 

where ExGr2'^(x) is the expectation of uj given by ^ //co'(x) dxdy and /(x) 
is the distance transform. The new energy to be minimized is composed of three 
terms and obtained by modifying the AT energy in (1) in two aspects. 

Firstly, the interaction term of (1) is additively augmented with (Exg r^'^Cx))^. 
This new term forces the minimizer to acquire a low average value with the aver- 
age being computed over the entire domain. At a first glance, this seems to favor 
spatial homogeneity by forcing the minimizer to attain values close to zero. Yet, 
the minimum of // (Ey^^QUj{x.))'^ dxdy is also reached when co oscillates, that 
is, when it attains both negative and positive values adding up to 0. In this 
respect, the third term is a separation term partitioning i? into subdomains of 
opposing signs. Due to the influence of the |V.p term which penalizes spatial 
inhomogeneities, locations of identical sign tend to form spatial groups. Obvi- 
ously, the minimizer of JJ^ |Va;(x)p + (ExGi7^(x))^ dxd^/ subject to homoge- 
nous Dirichlet boundary condition is the flat function oj = unless accompanied 
by an external inhomogeneity. 

Indeed, the purpose of the second modiflcation is to influence spatial group- 
ing of positive and negative values of in a particular way that the sign change 
separates the gross structure from the boundary detail. In particular, the up- 
per zero level set {i7+ = (x^y) G Q : uj{x^y) > 0} covers central regions whereas 
the lower zero level set {i7_ = {x^y) e f2 : uj{x^ y) < 0} covers peripheral regions 
containing limbs, protrusions and boundary texture or noise. Towards this end, 
the indicator Xni'^) is replaced by a weighted indicator that is a monotonically 
increasing function of the shortest distance to the boundary, namely, the distance 
transform. As before, the first term favors separation of the domain into phases; 
however, the phases are the level curves of the distance transform. Since, how- 
ever, the level curves of the AT function in (1) are equivalent to the level curves 
of a smooth distance transform [28], this change merely scales u without quali- 
tatively aff'ecting the geometry of its level curves. Nevertheless, when the terms 
considered together, the minimizer tends to have positive values at central loca- 
tions and negative values at peripheral locations because the penalty incurred 
by assigning negative values to central locations with higher positive / values is 
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higher than the penalty incurred by assigning negative values to locations with 
lower / values. 

Similar to the AT function, the new minimizer is a compromise between inho- 
mogeneity and homogeneity though the inhomogeneity is forced both externally 
(by /) and internally (by the third term); or it is the best approximation of an 
external inhomogeneity / subject to internal constraints. 

The parameter p should be chosen large enough so that the attachment to 
the external inhomogeneity should not dominate over the tendency to interact. 
Indeed, in the absence of the third term, a good practice is to chose p at least on 
the order of the maximum thickness for the diffusive effect of |V.p to influence 
the entire shape ([2]; Fig. 1 in [28]). The same argument also holds here since 
the effect of the third term is to partition Q into subdomains within which uj 
is morphologically similar to the AT function. Additionally, notice that the ex- 
pression responsible for sign change, // Co'(x) dxdy^ has been already normalized 
by As such, p should be larger than Y^|i7|. 

In Fig. 1 (a), an illustration for a 1-D case is given: uo is plotted for four differ- 
ent values of p ranging between ^J\^\ and 0.5 * |i7|. Naturally, uj gets flatter as p 
increases. (The flattening can be avoided by scaling either / or uj.) Nevertheless, 
the locations of the extrema and the zero crossings remain the same unless p is 
significantly smaller than Y^|i7|. Similarly in 2-D, the geometry of the level curves 
is stable as long as p is chosen suitably large. Illustrative level curves are de- 
picted in Fig. 1 (b-c). Absolute values of uj separately normalized within regions 
of identical sign are used for convenience of color visualization. Zero level curves 
separate central and peripheral structures in the form of upper and lower zero 
level sets: {i7+ = (x, ?/) G i? : uj{x^y) > 0} and {i7_ = (x,y) G i? : uoix^y) < 0}. 
The peripheral structure includes all the detail: limbs, protrusions, and bound- 
ary texture or noise. In contrast, the central structure is a very coarse blob-like 
form; it can even be thought as an interval estimate of the center whereas the 
centroid is the point estimate. 







'"^^ jlllllliir, ~ ______ ' ,\ 



(a) (b) (c) 

Fig. 1. (a) UJ for an interval for varying values of p ranging between and 
0.5 * |i7|. (b) Illustrative level curves of uj. 
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Most commonly, i?+ is a simply connected set. Of course, it may also be 
either disconnected or multiply connected. For instance, it is disconnected for 
a dumbbell-like shape (two blobs of comparable radii combined through a thin 
neck) whereas it is multiply connected for an annulus formed by two concentric 
circles. Indeed, the annulus gets split into three concentric rings where the middle 
ring is the i7+. For quite a many shapes, however, i?+ is a simply connected set. 

Firstly, shapes obtained by protruding a blob as well as shapes whose pe- 
ripheral parts are smaller or thinner than their main parts always have a simply 
connected i7+. This is expected: When the width of a part is small, the highest 
value of / inside the part is small. That is, the local contribution to (u; — /)^ 
incurring due to negative values is less significant for such a part as compared 
to locations with higher positive values of /. Consequently, uj tends to attain 
negative values on narrow or small parts as well as on protrusions. Shapes with 
holes also have a simply connected i?+ as long as the holes are far from the 
center. 

Secondly, even a dumbbell- like shape may have a simply connected i7+. This 
happens if the join area, namely, the neck is wide enough. Nevertheless, this does 
not cause any representational instability: Whereas the i7+ for a blob-like shape 
has a unique maximum located roughly at its centroid, the i7+ for a dumbbell- 
like shape has two local maxima indicating two bodies. Each body is captured 
by a connected component of an upper level set whose bounding curve passes 
through a saddle point. At a saddle point p, such that cj(p) = s, the s — level 
curve has a double point singularity, i.e. it forms a cross. As such, the upper level 
set {Q^ = {x^y) G i7+ : uj{x^y) > s} yields two disjoint connected components 
capturing the two parts of the central structure. 

In contrast to i7+, the peripheral structure i7_ is often multiply connected. 
Indeed, its hole(s) are carved by i7+. It is also possible that i?_ is disconnected. 
For instance, for an annulus, it is two concentric rings. Additionally, i7_ may be 
disconnected when there are several elongated limbs organized around a rather 
small central body, e.g., a palm tree. i7+, being small, is tolerated to grow and 
reach to the most concave parts of the shape boundary creating a split of i7_ 
by the zero-level curve. Similar to those in i7+, the level curves in i7_ that are 
passing through saddle points provide further partitioning. The partitions are in 
the form of lower level sets {i?^ = (x^y) G i7_ : uoix^y) < s}. 

To sum up, within both i7+ and i7_, nested open sets (upper level sets inside 
i7+ and lower level sets inside i7_) characterize the domain. The level curves 
bounding the level sets are either closed curves or closed curves with crossing 
points. The ones with crossing points are of particular interest because the re- 
spective level set is partitioned at those points into two distinct connected com- 
ponents. A crossing of a level curve occurs at a saddle point of uj. Of course, each 
lower level set may contain other saddle points. Consequently, the partitioning 
is binary and iterative and determined by the order of saddle points. 

It is not generically possible that a level curve has singular points of higher 
order because such singular points are unstable and may be removed by a slight 
change inoj. It is also highly unlikely that a connected component of an s — level 
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curve has two distinct crossing points. This issue is tackled in §3 via randomiza- 
tion. 

3 Randomized Hierarchy Tree 

Since the partitioning inside both the i7+ and i7_ of a shape are iterative and 
binary, the parts can be organized starting from the second level in the form 
of a proper binary tree. Let the shape be the root node and its children be 
the upper and lower zero level sets, namely, the disjoint regions of the central 
and peripheral structures. Suppose the central and peripheral structures are 
respectively composed of Nc and Np disjoint sets. Let us enumerate the nodes 
holding these sets as 11, 12, • • • , INc for the i7+ and as 21, 22, • • • , 2Np for the 
i7_ . This is the second level of the tree and the first level of the partitioning. Of 
course, the root may have more than two children. Nevertheless, starting from 
the children of the root, each subtree is a proper binary tree because all the splits 
inside an i?+ or i7_ occur at saddle points; that is, each connected component 
of the second level and its children either get split into two level sets or remain 
as they are. We call this hierarchical organization as the Initial Part Tree. A 
hypothetical initial part tree is illustrated in Fig. 2 (a). In a real example, the 
nodes hold application dependently selected attributes of the respective level 
sets. 

Binary splits according to saddle points produce collections of parts which 
are at the leaf level consistent across visual changes. However, the hierarchi- 
cal order and granularity of parts are not necessarily consistent. For instance, a 
weak saddle is easily removed when the shape is slightly smoothed. Likewise, cer- 
tain non-generic configurations such as level curves with spatially distinct saddle 
point singularities or triple point singularities cannot occur; indeed, such config- 
urations are easily replaced by one of the corresponding generic configurations 
which may differ for similar shapes. Furthermore, when a shape is occluded by 
another shape, added peripheral parts change the positions of some of the pre- 
vious level sets in the hierarchy. Nevertheless, the relative values of uj at saddle 
points prompting consecutive splits are stable indicators of the organizational 
hierarchy. Of course, attempting to convert a saddle point value to a tree depth 
by discretization brings back the previous robustness issue. 

Instead, we use the difference between the values of two successive saddle 
points as a measure of saliency for the partitioning prompted by the latter saddle 
point. Converting the saliency measure to a probability measure and considering 
probability measures for all nodes, we endow the initial part tree with a random 
structure from which possible re-organizations of the initial hierarchy tree are to 
be sampled. We call the new structure as the Randomized Part Hierarchy 
Tree. Below, we give the details of the randomization procedure. In contrast 
to the respective initial part tree, a random sample from a randomized part 
hierarchy tree is not necessarily a proper binary tree. 

The randomization starts from level 3 nodes and propagates through their 
children. Recall that this is the first level of nodes that are created via saddle 
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points. For each pair of siblings, there are two possible events: The pair of siblings 
either maintain their depth (no change in the local tree structure) or inherit the 
depth of their parent (change in the local tree structure). In the latter case, 
the node and its sibling replace their parent and become the children of their 
grandparent. The probabilities of the two events are derived from a quantity 
which we denote by Duj. It is a property of a split meaning that the Duj values 
of a pair of siblings are identical. Specifically, it is the difference between the 
saddle point values of a node and its parent divided by the saddle point value of 
the node. Because the magnitude of the saddle point value of a node is always 
greater than that of its parent, < Duj < 1; the equality is attained at level 3. 

A small value of Duo implies that the consecutive saddle points are closer in 
value; that is, a slight change in their value changes their order hence the local 
tree structure. Equivalently a large value of Duj implies that the consecutive 
saddle points are well separated, therefore, the local structure is stable. We re- 
quire that the probability p that a local structure change is necessary approaches 
1 as Duj approaches to the smallest possible value which is 0. Equivalently, p 
should approach as Duj approaches to its largest possible value. The function 
g-4Da; -g ^ good Candidate for estimating p; there is less than 2% chance for 
reorganization since = 0.018 for the largest possible Duj. 




Fig. 2. (a) An initial part tree (a) and its possible re-organizations (b-d). A 
random sample should be in one of the four forms. See the text. 



Let us consider the initial part tree in Fig. 2 (a). Assume that Duj = 0.301 
for nodes 211 and 212. With probability {1 — p) = 0.7, the local structure is 
preserved, whereas with probability p = 0.3 nodes 211 and 212 replace their 
parent and become children of their grandparent, the root. Assume that Duj = 
0.128 for 2111 and 2112. Then with probability (1 - g) = 0.4, the local structure 
is preserved, while with probability g = 0.6 nodes 2111 and 2112 replace their 
parent and become children of their grandparent which is either node 21 with 
(1— p) = 0.7 or the root with p = 0.3. Thus, there are four possible organizations: 
With probability (1 — p)(l — g) = 0.28, the entire structure is preserved. With 
probability (1 — p)q = 0.42, the tree is re-organized as in (b). With probability 
p(l — g) = 0.12, the tree is re-organized as in (c). With probability pq = 0.18, 
the tree is re-organized as in (d). 
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4 Experimental Results and Discussion 

To evaluate the effectiveness of endowing the part hierarchy tree with a prob- 
abilistic structure, we consider a pairwise matching problem. It is formulated 
as finding a maximal clique in the joint association graph of the pair of trees 
to be matched, e.g. [18]. At each experiment, each of the two randomized part 
hierarchy trees is independently sampled several times and then all the sample 
pairs are matched. The trees in the sample pair that yields the highest matching 
score are called as the winning re-organizations. 

Depending on the application, various properties related to the level sets 
stored at nodes can be used as node attributes. For instance, we extract parts 
enclosing each of the stored level sets and then use their area and the maximum 
uj values inside them as attributes. We remark that the maximum value of uj 
is related to the part width for the finest parts. Because boundaries of stored 
level sets pass through saddle points, an enclosing part is easily obtained as a 
morphologic watershed zone, whose seed is the respective level set [14]. Both to 
keep illustrations simple and resource requirements low, we require that each 
part to neighbor the central structure and each seed and part to have a certain 
size. Specifically, splits are performed only through the saddle points that reside 
on the boundaries of the watershed zones that are touching to the closure of the 
central structure and any split producing a seed that is less than 0.05% of the 
shape or a part that is less than 0.5% of the shape is ignored. 

We present four illustrative examples. In each case, correct associations are 
found despite several order and granularity inconsistencies resulting from oc- 
cluders, non- generic splits and weak saddles. 

The first example is a matching between a human silhouette and its occluded 
version. The winning re-organizations for each of the two trees are shown in 
Fig. 3. At each node, the watershed (enclosing part) is depicted as dark gray 
and the respective level set (seed part) depicted as black is superimposed on the 
part; the neighboring part (the light gray) is also shown even though it is not 
used for the matching. Due to page limits, we cannot provide the initial trees 
and probability distributions, but the numbering of the nodes already reveals 
the structure of the initial binary tree. 

Firstly, notice that the arms on the left and on the right reside at different lev- 
els in both of the initial trees as revealed by their respective five versus four digit 
node numbers. Ideally, the almost symmetric upper bodies (nodes 211) should 
contain two distinct saddle points pi and p2 such that cj(pi) = cj(p2) = <s; that 
is, two distinct saddle points on a single s — level curve should simultaneously 
yield the three nodes: 21111,2112 (arms) and 21112 (head). However, certain 
configurations including this one are not generic; even the slightest perturba- 
tion imposes a strict order on the saddle points. Thus, firstly, the combination 
of the head and either one of the arms is separated from the other arm then 
the head-arm combination is partitioned. Nevertheless, in each case, the saddle 
point value separating the head and arm on the left combination from the arm 
on the right is very close to the saddle point value separating the head from 
the arm on the left; e.g., for the first shape, the respective saddle point values 
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Fig. 3. The winning re-organizations for two shapes. The numbering of the nodes 
reveal their order in the initial binary tree. See the text. 



after normalization with respect to the global maximum of uo are —0.683 and 
—0.687 while the saddle point value separating the entire upper body from the 
entire lower body is —0.053. Clearly, the hierarchical order between the upper 
body and its children is more stronger than the hierarchical order among its 
children. We remark that even though the arm re-organization is not necessary 
for finding correct part correspondences since the structures of the upper bodies 
are already the same for the two initial trees, the left and right arms are brought 
to the same level. This is because the probabilities of retaining the initial binary 
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local structures is very low due to the closeness of the consecutive saddle point 
values. 

Secondly, notice that the legs of the occluded figure are at the sixth level 
whereas the legs of the un-occluded one are at the fourth level, as revealed 
by their node numbers. This is due to the infiuence of two additional parts 
(watershed regions) belonging to the occluder and poses a challenge for the tree 
matching. Nevertheless, the legs are brought to the same level as well as all of 
the corresponding parts and correct associations are found: 11 <^ 11 (central 
regions), 211 ^ 211 (upper bodies), 21111 ^ 21111 (arms on the left), 21112 4=^ 
21112 (heads), 2112 ^ 2112 (arms on the right), 212 ^ 21212 (lower bodies) 
2121 ^ 212121 (legs on the left) 2122 ^ 212122 (legs on the right). 

In the next three examples, due to limited space, only the matchings where 
at least one member of the matching pair is a leaf are depicted even though 
entire hierarchical structures are matched. Non-leaf nodes are circled. These 
examples also demonstrate the necessity of not restricting the correspondence 
search to leaf nodes. The matching between a cat and a horse in Fig. 4 illustrates 
a granularity inconsistency. Due to a weak saddle marked by the arrow in the 
left, the front legs of the horse are not further partitioned. Nevertheless, this 
inconsistency is resolved by matching the respective leaf node of the horse tree 
to a non-leaf node of the cat tree, the parent of the two nodes each holding a 
front leg of the cat. 




Fig. 4. A granularity inconsistency. Due to the weak saddle marked by the arrow 
in the left, the part of the horse corresponding to its front legs cannot be further 
partitioned. Nevertheless, it is correctly associated to a non-leaf node of the cat. 



Fig. 5 (a) depicts the matching of the same horse to another horse. In addition 
to the previous granularity inconsistency, there are several order inconsistencies 
which are not noticeable at the leaf level presentation. For instance, the rear 
body of the first horse firstly splits into the fourth leg and tail versus the third 
leg, and then the fourth leg is separated from the tail. On the other hand, the 
rear body of the second horse after a spurious division gets splits into the rear 
legs versus the tail, and then the two legs are separated. Consequently, a two 
level difference between the third leg of the first horse (node 2121) and the third 
leg of the second horse (node 212221) is formed. Despite both granularity and 
order inconsistencies, all of the parts are correctly matched. 
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(b) 


(a) 



Fig. 5. Two more cases involving both level and granularity inconsistencies. 

The final example (Fig. 5 (b)) involves several difficulties due to three weak 
saddles resulting with three unintuitive partitions for the second cat: Firstly, 
its head is fragmented; secondly, its rear body goes through a spurious division 
causing an erroneous shift in the levels of its sub-parts; thirdly, its fourth leg 
and tail are not separated. Nevertheless, the selected clique contains all of the 
correct associations. The rear body and its parts for the second cat are properly 
lifted one level up; consequently, the correct associations of the parts of the rear 
bodies are found successfully. The head of the first cat matches to the parent 
of the two leaves holding two unintuitive parts of the head of the second cat. 
The two head fragments of the second cat as well as the fourth leg and the tail 
of the first cat are correctly excluded from the selected clique as there are no 
corresponding parts in the other tree. 
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